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Abstract 

The paper introduces new bounds on the asymptotic density of parity-check matrices and the 
achievable rates under ML decoding of binary linear block codes transmitted over memoryless 
binary-input output-symmetric channels. The lower bounds on the parity-check density are 
expressed in terms of the gap between the channel capacity and the rate of the codes for which 
£N) ■ reliable communication is achievable, and are valid for every sequence of binary linear block 

codes. The bounds address the question, previously considered by Sason and Urbanke, of how 
r**"- ■ sparse can parity-check matrices of binary linear block codes be as a function of the gap to 

■ capacity. The new upper bounds on the achievable rates of binary linear block codes tighten 

previously reported bounds by Burshtein et al, and therefore enable to obtain tighter upper 
bounds on the thresholds of sequences of binary linear block codes under ML decoding. The 
bounds are applied to low-density parity-check (LDPC) codes, and the improvement in their 
tightness is exemplified numerically. 

■ 

X 

^ . 1 Introduction 

Error correcting codes which employ iterative decoding algorithms are now considered 
state of the art in the field of low-complexity coding techniques. These codes closely 
approach the capacity limit of many standard communication channels under iterative 
decoding. In jl], Khandekar and McEliece have suggested to study the encoding and 
decoding complexities of ensembles of iteratively decoded codes as a function of their 
gap to capacity. They conjectured that if the achievable rate under message-passing 
iterative (MPI) decoding is a fraction 1 — e of the channel capacity, then for a wide 
class of channels, the encoding complexity scales like In i and the decoding complexity 
scales like jln~. The only exception is the binary erasure channel (BEC) where the 
decoding complexity behaves like In ~ (same as encoding complexity) because of the 
absolute reliability of the messages passed through the edges of the graph (hence, every 
edge can be used only once during the iterative decoding process). 

Low-density parity-check (LDPC) codes are efficiently encoded and decoded due to 
the sparseness of their parity-check matrices. In his thesis j2] , Gallager proved that right- 
regular LDPC codes (i.e., LDPC codes with a constant degree (or) of the parity-check 
nodes) cannot achieve the channel capacity on a BSC, even under optimal ML decoding. 



This inherent gap to capacity is well approximated by an expression which decreases 
to zero exponentially fast in or. Richardson et al. ^Uj have extended this result, and 
proved that the same conclusion holds if ap, designates the maximal right degree. Sason 
and Urbanke later observed in that the result still holds when considering the average 
right degree. Gallager's bound Theorem 3.3] provides an upper bound on the rate of 
right-regular LDPC codes which achieve reliable communications over the BSC. Burshtein 
et al. have generalized Gallager's bound for general ensembles of LDPC codes transmitted 
over memoryless binary-input output-symmetric (MBIOS) channels P, and the work in 
[T2*] relies on their generalization. 

Consider the number of ones in a parity-check matrix which represents a binary 
linear code, and normalize it per information bit (i.e., with respect to the dimension 
of the code). This quantity (which will be later defined as the density of the parity- 
check matrix) is equal to times the average right degree of the bipartite graph that 
represents the code, where R is the rate of the code in bits per channel use. In |12j . 
Sason and Urbanke considered how sparse can parity-check matrices of binary linear 
block codes be, as a function of their gap to capacity (where this gap depends in general 
on the channel and on the decoding algorithm). An information-theoretic lower bound 
on the asymptotic density of parity-check matrices was derived in ^21 Theorem 2.1] 
where this bound applies to every MBIOS channel and every sequence of binary linear 
block codes achieving a fraction 1 — e of the channel capacity with vanishing bit error 
probability. It holds for an arbitrary representation of full-rank parity-check matrices 
for these codes, and is of the form Kl+ ^ ln g w here K\ and K% are constants which only 
depend on the channel. Though the logarithmic behavior of this lower bound is in essence 
correct (due to a logarithmic behavior of the upper bound on the asymptotic parity-check 
density in ^21 Theorem 2.2]), the lower bound in ^21 Theorem 2.1] is not tight (with the 
exception of the BEC, as demonstrated in 112, Theorem 2.3], and possibly also the binary 
symmetric channel (BSC)). The derivation of the bounds in this paper was motivated by 
the desire to improve the results in ^ Theorems 1 and 2] and [T2*l Theorem 2.1] which 
are based on a two-level quantization of the log-likelihood ratio (LLR). They provide 
upper bounds on the achievable rates of LDPC codes over MBIOS channels, and lower 
bounds on their asymptotic parity-check density. 

In j2] , Measson and Urbanke derived an upper bound on the thresholds under maximum- 
likelihood (ML) decoding of LDPC ensembles when the codes are transmitted over the 
BEC. Their general approach relies on EXIT charts, having a surprising and deep con- 
nection with the maximum a posteriori (MAP) threshold due to the area theorem for 
the BEC. Generalized extrinsic information transfer (GEXIT) charts were recently intro- 
duced by Measson, Montanari, Richardson and Urbanke 0; they form an extension of 
EXIT charts to general MBIOS channels, and in particular, they fulfill the area theorem 
(see fTT] Section 3.4.10]). This conservation law enables one to get upper bounds on the 
thresholds of turbo-like ensembles under bit-MAP decoding. The bound was shown to 
be tight for the BEC [5], and was conjectured to be tight for general MBIOS channels. 

A new method for analyzing LDPC codes and low-density generator-matrix (LDGM) 
codes under bit MAP decoding is introduced by Montanari in [7j. The method is based 
on a rigorous approach to spin glasses, and allows a construction of lower bounds on the 
entropy of the transmitted message conditioned on the received one. The calculation 
of this bound is rather complicated, and its complexity grows exponentially with the 
maximal right and left degrees (see [3 Eqs. (6.2) and (6.3)]); this imposes a considerable 
difficulty in its calculation (especially, for continuous-output channels). Since the bounds 



in [3[7] are derived for ensembles of codes, they are probabilistic in their nature; based on 
concentration arguments, they hold asymptotically in probability 1 as the block length 
goes to infinity. Based on heuristic statistical mechanics calculations, it was conjectured 
that the bounds in \7], which hold for general LDPC and LDGM ensembles over MBIOS 
channels, are tight. 

We present in this paper new bounds on the achievable rates and the asymptotic 
parity-check density of sequences of binary linear block codes. The bounds in and 
this paper are valid for every sequence of binary linear block codes, in contrast to a high 
probability result which was previously derived for the binary erasure channel (BEC) 
from density evolution analysis jT3|. Shokrollahi proved in |T3] that when the codes are 
communicated over a BEC, the growth rate of the average right degree (i.e., the average 
degree of the parity-check nodes in a bipartite Tanner graph) is at least logarithmic in 
terms of the gap to capacity. The statement in ^3] is a high probability result, and 
hence it is not necessarily satisfied for every particular code from this ensemble. Further, 
it assumes a sub-optimal (iterative) decoding algorithm, where the statements in [TJ [TJ] 
and this paper are valid even under optimal ML decoding. 

In this paper, preliminary material is presented in Section |2l and the theorems are 
introduced in Section EH Numerical results are exemplified and explained in Section 0] 
Finally, we summarize the discussion in Section |3] and present interesting issues which 
deserve further research. The interested reader is referred to the full paper version 
which includes the proofs of the theorems in Sectional discussions and some additional 
numerical results which were omitted here because of space limitations. 

We note that the statements in this paper refer to the case where the parity-check 
matrices are full rank. Though it seems like a mild requirement for specific linear codes, 
this poses a problem when considering ensembles of LDPC codes. In the latter case, a 
parity-check matrix, referring to a randomly chosen bipartite graph with a given pair of 
degree distributions, may not be full rank. Fortunately, as we later explain in this paper 
(see Section EJ), the statements still hold for ensembles when we replace the code rate 
with the design rate. 

2 Preliminaries 

We introduce here some definitions and theorems from PIE] which serve as a preliminary 
material for the rest of the paper. Definitions 12.11 and 12.21 are taken from [T2*l Section 2]. 

Definition 2.1 (Capacity- Approaching Codes). Let {C m } be a sequence of codes 
of rate R m , and assume that for every m, the codewords of the code C m are transmitted 
with equal probability over a channel whose capacity is C. This sequence is said to 
achieve a fraction 1 — e of the channel capacity with vanishing bit error probability if 
linim^oo Rm = (1 — e)C, and if there exists a decoding algorithm under which the average 
bit error probability of the code C m tends to zero in the limit where m — > oo. 

Definition 2.2 (Parity-Check Density). Let C be a binary linear code of rate R and 
block length n, which is represented by a parity-check matrix H. We define the density 
of H, call it A = A(H), as the normalized number of ones in H per information bit. The 
total number of ones in H is therefore equal to nRA. 

Definition 2.3 (Log-Likelihood Ratio (LLR)). Let Py\x(-\-) be the conditional pdf 
of an arbitrary MBIOS channel. The log-likelihood ratio (LLR) at the output of the 



channel is 

\mx(y\x = i) 

Throughout the paper, we assume that all the codewords of a binary linear block 
code are equally likely to be transmitted. Also, we use the notation /i2(") for the binary 
entropy function to base 2, i.e., h 2 (-) = — xlog 2 (x) — (1 — x) log 2 (l — x). 

Theorem 2.1 (An Upper Bound on the Achievable Rates for Reliable Com- 
munication over MBIOS Channels). [1, Theorem 2]: Consider a sequence {C m } of 
binary linear block codes of rate R m , and assume that their block length tends to infinity 
as m —>■ oo. Let H m be a full-rank parity-check matrix of the code C m , and assume that 
dk,m designates the fraction of the parity-check equations involving k variables. Let 



dk = liminf dk, m , R — liminf R m . (1) 

m— >oo 



Suppose that the transmission of these codes takes place over an MBIOS channel with 
capacity C bits per channel use, and let 



COO 

tr = - I mxn(f(y),f(-y)) dy (2) 
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— oo 



where f(y) = py\x(y\X = 1) designates the conditional pdf oi the output of the MBIOS 
channel. Then, a necessary condition for vanishing block error probability as m —>■ oo is 

R<1 l ~ C 



Theorem 2.2 (Lower Bounds on the Asymptotic Parity-Check Density with 
2-Levels Quantization). [T21 Theorem 2.1]: Let {C m } be a sequence of binary linear 
block codes achieving a fraction 1 — s of the capacity of an MBIOS channel with vanishing 
bit error probability. Denote A m as the density of a full-rank parity-check matrix of the 
code C m . Then, the asymptotic density satisfies 

liminfA m >^ — e - (3) 

m— >oo 1 — £ 



where 



K _ (i - c) !g (jm K - Li£ M) 



and w is defined in (J2J). For a BEC with erasure probability p, the coefficients K\ and 
K 2 in Q are improved to 



Using standard notation, an ensemble of (n, A, p) LDPC codes is characterized by its 
length n, and the polynomials A(x) = A^* -1 and p(x) = Y^L 2 Pi x% ~ 1 ^ wnere \ 

(pi) is equal to the probability that a randomly chosen edge is connected to a variable 
(parity-check) node of degree i. The variables (parity-check sets) are represented by the 
left (right) nodes of a bipartite graph which represents an LDPC code. The design rate 
of this ensemble is given by 

Jo 1 p(x) dx 



lo dx 



3 Main Results 



3.1 Approach I: Bounds Based on Quantization of the LLR 

In this section, we introduce bounds on the achievable rates and the asymptotic parity- 
check density of sequences of binary linear block codes. The bounds generalize previously 
reported results in pQ and jT5j which were based on a symmetric two-level quantization of 
the LLR. This is achieved by extending the concept of quantization to an arbitrary integer 
power of 2; to this end, the analysis relies on the Galois field GF(2 d ). We commence by 
deriving a lower bound on the conditional entropy of a transmitted codeword given the 
received sequence. 

Proposition 3.1. Let C be a binary linear block code of length n and rate R. Let 
x = (xi, . . . x n ) and y = (yi, . . . , y n ) designate the transmitted codeword and received 
sequence, respectively, when the communication takes place over an MBIOS channel with 
conditional pdfpy\x(-\-)- For an arbitrary d > 2 and < Z 2 d-i_i < . . . < / 2 < h < lo — 00, 
let us define the set of probabilities {psjlj^ 1 as follows: 

f Pr{/ S+1 < LLR(y) < l s I X = 0} s = 0, . . . , 2 d ~ 1 - 2 

Pr{0 < LLR(y) < Z 2d -i_! I X = 0} + | Pr{LLR(y) = | X = 0} s = 2^ - 1 
Pr{-Z 2d - 1 _ 1 < LLR(F) < I X = 0} + \ Pr{LLR(F) = | X = 0} s = 2 d ~ l 
{ Pr{-Z 2Ms+1) < LLR(F) < -l 2i _ s I X = 0} s = 2 d ~ l + 1, . . . , 2 a 



For an arbitrary full-rank parity-check matrix of the code C, let dk denote the fraction of 
the parity-checks involving k variables. Then, the conditional entropy of the transmitted 
codeword given the received sequence satisfies 



k k k ,...,k 2d _ 1 _ 1 



i=0 




2 d-i_i 



1- n 1- 



i=0 



2p 2 d„ 



l-i 



Pi+P2d-1- 



Theorem 3.1 ("2 d -Level Quantization" Lower Bound on the Asymptotic Par- 
ity-Check Density of Binary Linear Block Codes). Let {C m } be a sequence of 
binary linear block codes achieving a fraction 1 — e of the capacity of an MBIOS channel 



with vanishing bit error probability. Denote A m as the density of a full-rank parity-check 
matrix of the code C m . Then, the asymptotic density satisfies 



lim inf A m > 

m— >oo 



Ki + K 2 ln± 



where 



K x = K 2 In 



1-C 



21n(2) C 



1 - C 



2 <J - 1 -l 



i=0 



(Pi - ggd-j-i^ 



Here, d > 2 is an arbitrary integer and the probabilities {p{\ are introduced in (JUJ) in 
terms of l\ > . . . > ^^-l £ The optimal vector of quantization levels (li, . . . , £ 2 d-i_i) 
is given implicitly by the set of 2 d ~ l — 1 equations 



P2"-l-i + e P 



(pi +p 2 i-i~i) 2 (p»-i 
where such a solution always exists. 



1,...,2 



d-l 



1. 



Corollary 3.1 ('^^-Level Quantization" Upper Bound on the Asymptotic Achiev- 
able Rates of Sequences of Binary Linear Block Codes). Let {C m } be a sequence 
of binary linear block codes whose codewords are transmitted with equal probability over 
an MBIOS channel, and suppose that the block length of this sequence of codes tends 
to infinity as m — > oo. Let dk, m be the fraction of the parity-check nodes of degree k in 
an arbitrary representation of the code C m by a bipartite graph which corresponds to a 
full-rank parity-check matrix. Then a necessary condition for this sequence to achieve 
vanishing bit error probability as m — ► oo is that the asymptotic rate R of this sequence 
satisfies 



R < 1 - max { (1 - C) 




^ 2 d-i_i 

■ i ki=k 



k 



V2<i- 



i=0 



ho 



n 

i=0 



l-i 



Pi + pa*_i. 



2 d -l 

2 p* 

i=2 d ~ 1 



2 d -l 

k i=2 d - 1 



where d > 2 is arbitrary, the probabilities {pi} are introduced in ©, and c4 and i? are 
introduced in (JTJ). 



3.2 Approach II: Bounds without Quantization of the LLR 

Similarly to the previous section, we derive bounds on the asymptotic achievable rate 
and the asymptotic parity-check density of an arbitrary sequence of binary, linear block 
codes transmitted over an MBIOS channel. As in Section ETT1 the derivation of these two 
bounds is based on a lower bound on the conditional entropy of a transmitted codeword 
given the received sequence at the output of an arbitrary MBIOS channel. 



Proposition 3.2. Let C be a binary linear code of length n and rate R transmitted over 
an MBIOS channel. Let x = (xi, x 2 , ■ ■ ■ , x n ) and y = (y 1 ,y 2 , . . . , y n ) be the transmitted 
codeword and the received sequence, respectively. For an arbitrary representation of the 
code C by a full-rank parity-check matrix, let dk designate the fraction of the parity-check 
equations of degree k. Then the conditional entropy of the transmitted codeword given 
the received sequence satisfies 



H(X.\Y) 



n 



> 1-C-(1-R)l 1 ^-V-^ -V4 ( I™ a(l)(l + e- l )ta,nh 2p ( -] 



d 



where a(-) denotes the conditional pdf of the LLR given that the transmitted symbol is 
zero. 

Theorem 3.2 ("Un-Quantized" Lower Bound on the Asymptotic Parity-Check 
Density of Binary Linear Block Codes). Let {C m } be a sequence of binary linear 
codes achieving a fraction 1 — e of the capacity C of an MBIOS channel with vanishing 
bit error probability. Denote A m as the density of a full-rank parity-check matrix of the 
code C m . Then, the asymptotic density satisfies 

Kjjx) + K 2 {x)\r\ 
hmmtA m > sup (7) 



x€(0,A] 



where 



1 - C ln \~cr^J K , , 1 C 



g (i-c) 



and 



C In (I) ' — C In (I) 



Jo 1 + e-' ' 1 Ikk otherwise 



21n(2) 

Remark 3.1. For a BEC with erasure probability p, the maximization of the RHS of 
(|7j) yields that the maximal value is achieved for x = 1 — p when e < (otherwise, if 
e > the lower bound is useless as it becomes non-positive). Hence, the lower bound 
on the asymptotic parity-check density stated in Theorem 13.21 coincides with the bound 
for the BEC in [T^l Eq. (3)]. This lower bound was demonstrated in |12 S Theorem 2.3] to 
be tight. This is proved by showing that the sequence of right-regular LDPC ensembles of 
Shokrollahi 13, is optimal in the sense that it achieves (up to a small additive coefficient) 
the lower bound on the asymptotic parity-check density for the BEC. 

Remark 3.2. The lower bound on the parity-check density in Theorem 13 .21 is uniformly 
tighter than the one in ^21 Theorem 2.1] (except for the BSC and BEC where they 
coincide). For a proof of this claim, the reader is referred to [TH Appendix D.2]. 

Corollary 3.2 ("Un-Quantized" Upper Bound on the Asymptotic Achievable 
Rates of Sequences of Binary Linear Block Codes). Let {C m } be a sequence of 
binary linear block codes whose codewords are transmitted with equal probability over 
an MBIOS channel, and assume that the block lengths of these codes tend to infinity 
as m —>■ oo. Let (4 jm be the fraction of the parity-check nodes of degree k for arbitrary 
representations of the codes C m by bipartite graphs which corresponds to a full-rank 



parity-check matrix. Then a necessary condition on the achievable rate (R) for obtaining 
vanishing bit error probability as m — > oo is 



1 L_yi_J zYd k ( r ) ail) (l + e~ l ) tanh 2p f-^| dl) \ 

where d k and R are introduced in ((H). 



4 Numerical Results 

We present here numerical results for the information-theoretic bounds in Sectional As 
expected, they significantly improve the numerical results presented in P3 Section 4] and 
[T2*l Section 4]. This improvement is attributed to the fact that, in contrast to PIE], in 
the derivation of the bounds in this paper, we do not perform a two-level quantization 
of the LLR which in essence converts the arbitrary MBIOS channel (whose output may 
be continuous) to a BSC. Throughout this section, we assume transmission of the codes 
over the binary-input AWGN channel. 

We note that the statements in Sections 121 and H3 refer to the case where the parity- 
check matrices are full rank. Though it seems like a natural requirement for specific 
linear codes, this poses a problem when considering ensembles of LDPC codes. In the 
latter parity-check matrix, referring to a randomly chosen bipartite graph with 

a given pair of degree distributions, may not be full rank. Considering ensembles of 
LDPC codes, it follows from the proofs in Sections 3 and 4] that the statements 
stay valid with the following modifications: the actual code rate R of a code which is 
randomly picked from the ensemble is replaced by the design rate (Rd) of the ensemble, 
and becomes the degree distribution of the parity-check nodes referring to the 

original bipartite graph which represents a parity-check matrix, possibly not of full rank. 
By doing this modification, the bound becomes looser when the asymptotic rate of the 
codes is strictly above the design rate of the ensemble. In light of this modification, 
we note that the fraction c4 of nodes of degree k is calculated in terms of the degree 
distribution p(-) by the equation 

a - P k 1 

— — — j . 

k J Q p( x ) dx 

This conclusion about the possible replacement of the code rate with the design rate in 
case that the parity-check matrices are not full rank was also noted in QjJ p. 2439]. 



4.1 Thresholds of LDPC Ensembles under ML Decoding 

Table Q provides bounds on the thresholds of Gallager's regular LDPC ensembles under 
ML decoding. It also gives an indication on the inherent loss in performance due to the 
sub-optimality of MPI decoding. 

The bounds on the achievable rates in pQ and Corollaries 13.11 and 13.21 provide lower 
bounds on the |^ thresholds under ML decoding. For Gallager's regular LDPC ensem- 
bles, the gap between the thresholds under ML decoding and the exact thresholds under 
the sum-product decoding algorithm (calculated using density-evolution analysis) are 



LDPC 


Capacity 
Limit 


2-Level 
Bound 


4-Level 
Bound 


8-Level 
Bound 


Un-Quantized 
Lower Bound 


Upper 
Bound [3j 


DE 
Threshold 


(3,6) 


+0.187 dB 


+0.249 dB 


+0.332 dB 


+0.361 dB 


+0.371 dB 


+0.673 dB 


1.110 dB 


(4,6) 


-0.495 dB 


-0.488 dB 


-0.472 dB 


-0.463 dB 


-0.463 dB 


-0.423 dB 


1.674 dB 


(3,4) 


-0.794 dB 


-0.761 dB 


-0.713 dB 


-0.694 dB 


-0.687 dB 


-0.510 dB 


1.003 dB 



Table 1: Comparison of thresholds for Gallager's ensembles of regular LDPC codes trans- 
mitted over the binary-input AWGN channel. The 2-level lower bound on the threshold 
of jP- refers to ML decoding, and is based on [TJ Theorem 1] (see also O Table II]). The 
4-level, 8-level and un-quantized lower bounds apply to ML decoding, and are based on 
Corollaries 13.11 and 13.21 The upper bound on the threshold of holds under 'typical 
pairs' decoding [3 a (and hence, also under ML decoding), and the density evolution (DE) 
thresholds are based on density evolution for MPI decoding 

rather large. For this reason, we also compare the lower bounds on the thresholds 
under ML decoding with upper bounds on the thresholds which rely on "typical pairs 
decoding" [Hj; an upper bound on the thresholds under an arbitrary sub-optimal de- 
coding algorithm (e.g., "typical pairs decoding") also forms an upper bound on these 
thresholds under optimal ML decoding. It is shown in Tabled that the gap between the 
thresholds under iterative decoding and the bounds for ML decoding (see the columns 
referring to the DE threshold and the upper bound based on "typical pairs decoding") 
is rather large. This is attributed to the sub-optimality of belief propagation decoding 
for regular LDPC ensembles. On the other hand, it is also demonstrated in Table [I] that 
the gap between the upper and lower bounds on the thresholds under ML decoding is 
much smaller. For example, according to the numerical results in Table ^ the inherent 
loss in the asymptotic performance due to the sub-optimality of belief propagation for 
Gallager's ensemble of (4, 6) regular LDPC codes (whose design rate is | bits per channel 
use) ranges between 2.097 and 2.137 dB. 

Similarly to Tabled numerical results referring to ensembles of irregular LDPC codes 
which closely approach the capacity limit over the binary-input AWGN channel are shown 
in [TU Section 5]. 

The plots in Figure Q compare different lower bounds on the -^-threshold under ML 
decoding of right-regular LDPC ensembles. The plots refer to a right degree of 6 (left plot) 
or 10 (right plot). The following lower bounds are depicted in these plots: the Shannon 
capacity limit, the 2-level quantization lower bound in PjJ Theorem 1], the 4 and 8-level 
quantization bounds of the LLR in Section 13. 1| and finally, the bound in Section 13.21 
where no quantization of the LLR is performed. It can be observed from the two plots 
in Figure Q that the range of code rates where there exists a visible improvement with 
the new lower bounds depends on the degree of the parity-check nodes. In principle, the 
larger the value of the right-degree is, then the improvement obtained by these bounds 
is more pronounced starting from a higher rate code rate (e.g., for a right degree of 6 
or 10, the improvement obtained by the new bounds is observed for code rates starting 
from 0.35 and 0.55 bits per channel use, respectively). 

4.2 Lower Bounds on the Asymptotic Parity-Check Density 

The lower bound on the parity-check density derived in Theorem 13.21 enables to assess 
the tradeoff between asymptotic performance and asymptotic decoding complexity (per 



Right-Regular LDPC a R = 6 Right-Regular LDPC a R = 10 




Figure 1: Comparison between different lower bounds on the threshold under ML decod- 
ing for right-regular LDPC ensembles with or = 6 (left plot) and <2r = 10 (right plot). 
The transmission takes place over the binary-input AWGN channel. 



iteration) of an MPI decoder. This bound tightens the lower bound on the asymptotic 
parity-check density derived in [T2| Theorem 2.1]. Fig. |2] compares these bounds for 
codes of rate | (left plot) and | (right plot) where the bounds are plotted as a function 
of It can be observed from Fig. [21 that as increases, the advantage of the bound 
in Theorem 13.21 over the bound in ^21 Theorem 2.1] diminishes. This follows from the 
fact that as the value of is increased, the two-level quantization of the LLR used in pQ 
and [12, Theorem 2.1] better captures the true behavior of the MBIOS channel. It is also 
reflected in this figure that as e tends to zero (i.e., when the gap to capacity vanishes), 
the slope of the bounds becomes very sharp. This is due to the logarithmic behavior of 
the bounds. 




0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.6 1.8 2 2.2 2.4 2.6 2.8 
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Figure 2: Comparison between lower bounds on the asymptotic parity-check density of 
binary linear block codes transmitted over a binary-input AWGN channel. The dashed 
line refers to [T2"l Theorem 2.1], and the solid line refers to Theorem 13.21 The left and 
right plots refer to code rates of | and |, respectively. The Shannon capacity limit for 
these code rates corresponds to of 0.187 dB and 1.626 dB, respectively. 



5 Summary and Outlook 



We present improved lower bounds on the asymptotic density of parity-check matrices 
and upper bounds on the achievable rates of binary linear block codes transmitted over 
memoryless binary-input output-symmetric (MBIOS) channels. The improvements are 
w.r.t. the bounds given in PIE]- The information-theoretic bounds are valid for every 
sequence of binary linear block codes, in contrast to high probability results which follow 
from probabilistic analytical tools (e.g., density evolution (DE) analysis under iterative 
decoding). The bounds hold under optimal ML decoding, and hence, they hold in par- 
ticular under any sub-optimal decoding algorithm. We apply the bounds to ensembles 
of low-density parity-check (LDPC) codes. The significance of the bounds is the follow- 
ing: firstly, by comparing the new upper bounds on the achievable rates with thresholds 
provided by DE analysis, we obtain rigorous bounds on the inherent loss in performance 
of various LDPC ensembles. This degradation in the asymptotic performance is due to 
the sub-opt imality of MPI decoding (as compared to optimal ML decoding). Secondly, 
the parity-check density can be interpreted as the complexity per iteration under MPI 
decoding. Therefore, by tightening the reported lower bound on the asymptotic parity- 
check density Theorem 2.1], the new bounds provide better insight on the tradeoff 
between the asymptotic performance and the asymptotic decoding complexity of itera- 
tively decoded LDPC codes. Thirdly, a new lower bound on the bit error probability 
of binary linear block codes presented in Corollary 4.2] tightens the reported lower 
bound in ^21 Theorem 2.5]. 

The comparison between the quantized and un-quantized bounds gives insight on the 
effect of the number of quantization levels of the LLR (even if they are chosen optimally) 
on the achievable rates, as compared to the ideal case where no quantization is done. The 
results of such a comparison are shown in Table ^ and indicate that the improvement 
in the tightness of the bounds when more than 8 levels of quantization are used (in case 
the quantization levels are optimally determined) is marginal. 

The bounds on the thresholds of LDPC ensembles under optimal ML decoding depend 
only on the degree distribution of their parity-check nodes and their design rate. For a 
given parity-check degree distribution (p) and design rate (R), they provide an indication 
on the inherent gap to capacity which is independent of the choice of A (as long as the 
pair of degree distributions (A, p) yield the design rate R). Therefore, the bounds are not 
expected to be tight for LDPC ensembles with a given pair of degree distributions (A, p). 
The numerical results shown in Table ^ indicate, however, that these bounds are useful 
for assessing the inherent gap to capacity of LDPC ensembles. As a topic for further 
research, it is suggested to examine the possibility of tightening the bounds for specific 
ensembles by explicitly taking into account the exact characterization of A. 

The lower bound on the asymptotic parity-check density in ^21 Theorem 2.1] and its 
improvements in Section El grow like the log of the inverse of the gap (in rate) to capacity. 
The result in [T2"l Theorem 2.3] shows that a logarithmic growth rate of the parity-check 
density is achievable for Gallager's regular LDPC ensemble under ML decoding when 
transmission takes place over an arbitrary MBIOS channel. These results show that 
for any iterative decoder which is based on the representation of the codes by Tanner 
graphs, there exists a tradeoff between asymptotic performance and complexity which 
cannot be surpassed. Recently, it was shown in [S] that better tradeoffs can be achieved by 
allowing more complicated graphical models; for the particular case of the binary erasure 
channel (BEC), the encoding and the decoding complexity of properly designed codes 



on graphs remain bounded as the gap to capacity vanishes. To this end, Pfister, Sason 
and Urbanke consider in jH| Theorems 1 and 2] ensembles of irregular repeat-accumulate 
codes which involve punctured bits, and allow in this way a sufficient number of state 
nodes in the Tanner graph representing the codes. This surprising result is considered in 
[HJ Theorem 4] , by a derivation of an information-theoretic lower bound on the decoding 
complexity of randomly punctured codes on graphs whose transmission takes place over 
MBIOS channels. The approach for the derivation of the bounds in |HJ Theorem 4] relies 
on the analysis in ^21 Theorem 2.1]. It is suggested to tighten the lower bounds in (HI 
Theorem 4] by relying on the approach used to prove Theorem 13.21 in [T4*j (as compared 
to the derivation of the bound in (Hj Theorem 4] which relies on the analysis in [T2"l 
Theorem 2.1]). The interested reader is referred to [Tl], including proofs of the theorems, 
discussions and more numerical results that were omitted here for the sake of brevity. 

Acknowledgment 

The second author wishes to acknowledge Riidiger Urbanke for stimulating discussions 
during the preparation of the work in ^2] which motivated the research in this paper. 
The authors acknowledge Andrea Montanari for his correspondence regarding [7j. 

References 

[1] D. Burshtein, M. Krivelevich, S. Litsyn and G. Miller, "Upper bounds on the rate of LDPC codes," IEEE Trans, 
on Information Theory, vol. 48, no. 9, pp. 2437-2449, September 2002. 

[2] R. G. Gallager, Low-density parity-eheck codes, Cambridge, MA, USA, MIT Press, 1963. 

[3] H. Jin and R. J. McEliece, "Typical pairs decoding on the AWGN channel," Proceedings 2000 International Sym- 
posium on Information Theory and Its Applications, pp. 180—183, Honolulu, Hawaii, U.S.A., November 5—8, 2000. 

[4] A. Khandckar and R. J. McEliece, "On the complexity of reliable communications on the erasure channel," Pro- 
ceedings 2001 International Symposium on Information Theory, p. 1, Washington, DC, USA, June 24-29, 2001. 

[5] C. Measson and R. Urbanke, "An upper bound on the ML threshold of LDPC ensembles over the BEC," Proceed- 
ings of the Forty-First Allerton Conference on Communications, Control and Computing, pp. 478-487, Allerton, 
Monticello, USA, October 1-3, 2003. 

[6] C. Measson, A. Montanari, T. Richardson and R. Urbanke, "Life above threshold: From list decoding to area 
theorem and MSE,", 2004 IEEE Information Theory Workshop, San Antonio, TX, USA, October 24-29, 2004. 

[7] A. Montanari, "Tight bounds for LDPC codes and LDGM codes under MAP decoding," accepted to IEEE Trans, 
on Information Theory, 2005. [Online], Available: http://arxiv.org/abs/cs.IT/0407060 

[8] H. D. Pfister, I. Sason, and R. Urbanke, "Capacity-achieving ensembles for the binary erasure channel with bounded 
complexity," IEEE Trans, on Information Theory, vol. 51, no. 7, pp. 2352-2379, July 2005. 

[9] T. Richardson and R. Urbanke, "The capacity of low-density parity-check codes under message-passing decoding," 
IEEE Trans, on Information Theory, vol. 47, no. 2, pp. 599-618, February 2001. 

[10] T. Richardson, A. Shokrollahi and R. Urbanke, "Design of capacity-approaching irregular low-density parity-check 
codes," IEEE Trans, on Information Theory, vol. 47, no. 2, pp. 619—637, February 2001. 

[11] T. Richardson and R. Urbanke, Modern Coding Theory, to be published, Cambridge Press, June 2005. [Online]. 
Available: http://lthcwww.epfl . ch/mct/index .php 

[12] I. Sason and R. Urbanke, "Parity-check density versus performance of binary linear block codes over memoryless 
symmetric channels," IEEE Trans, on Information Theory, vol. 49, no. 7, pp. 1611-1635, July 2003. 

[13] A. Shokrollahi, "New sequences of time erasure codes approaching channel capacity," in Proceedings of the 13th 
International Symposium on Applied Algebra, Algebraic Algorithms and Error- Correcting Codes, Lectures Notes in 
Computer Science 1719, Springer Verlag, pp. 65-76, 1999. 

[14] G. Wiechman and I. Sason, "Improved bounds on the parity-check density and achievable rates of binary linear 
block codes with applications to LDPC codes," submitted to IEEE Trans, on Information Theory, May 2005. See 
http : //arxiv . org/abs/cs . IT/0505057 . 



